Feature: Basic Agentic Pick and Place - reimplemented manipulation skills #1237
Conversation
…lues have tiny errors that can get out of joint limit range
mustafab0
requested review from
JalajShuklaSS,
alexlin2,
leshy,
paul-nechifor and
spomichter
Greptile OverviewGreptile SummaryThis PR adds an agent-invokable skill layer to manipulation by converting The changes fit into the existing architecture by leaning on Confidence Score: 2/5
Important Files Changed
Sequence DiagramsequenceDiagram
autonumber
participant User as Human user
participant HI as human_input (CLI)
participant Agent as llm_agent
participant Skills as SkillCoordinator
participant Manip as ManipulationModule (SkillModule)
participant WM as WorldMonitor
participant Planner as RRTConnectPlanner
participant CC as ControlCoordinator
participant Task as Trajectory Task (JointTrajectoryController)
User->>HI: type "pick up the cup"
HI->>Agent: user message
Agent->>Skills: select skill + args
Skills->>Manip: pick(object_name)
Manip->>WM: refresh_obstacles(min_duration)
WM-->>Manip: obstacles added
Manip->>Manip: _generate_grasps_for_pick()
alt GraspingModule RPC wired
Manip->>Manip: get_rpc_calls("GraspingModule.generate_grasps")
Manip-->>Manip: grasp candidates (intended)
else fallback heuristic
Manip->>WM: list_cached_detections()
WM-->>Manip: detections snapshot
Manip-->>Manip: heuristic grasp pose
end
Manip->>Planner: plan_joint_path(start, goal)
Planner-->>Manip: JointPath
Manip->>Manip: generate JointTrajectory
Manip->>CC: task_invoke(task, "execute", {trajectory})
CC->>Task: execute(trajectory)
Task-->>CC: accepted
CC-->>Manip: result
loop wait for completion
Manip->>CC: task_invoke(task, "get_status" or "get_state")
CC->>Task: get_status()/get_state()
Task-->>CC: TrajectoryStatus / None
CC-->>Manip: status
end
Manip-->>Skills: streamed progress strings
Skills-->>Agent: tool results
Agent-->>HI: assistant response
|
![greptile-apps[bot]](https://avatars.githubusercontent.com/in/867647?s=60&v=4){kind=small}
| try: | ||
| generate = self.get_rpc_calls("GraspingModule.generate_grasps") | ||
| result_str = generate(object_name, object_id, True) | ||
| # If generate_grasps returned actual PoseArray via the grasps port, | ||
| # we need to get the poses. For now, check if it returned an error string. | ||
| if isinstance(result_str, str) and "No" in result_str: | ||
| logger.info(f"GraspGen returned: {result_str}, falling back to heuristic") | ||
| else: | ||
| # GraspGen succeeded — get poses from the grasps port or RPC | ||
| logger.info(f"GraspGen result: {result_str}") | ||
| # Try to get the grasp poses via RPC | ||
| try: | ||
| get_grasps = self.get_rpc_calls("GraspingModule.get_latest_grasps") | ||
| grasp_poses: PoseArray | None = get_grasps() | ||
| if grasp_poses and len(grasp_poses.poses) > 0: | ||
| return list(grasp_poses.poses) | ||
| except Exception: |
There was a problem hiding this comment.
Invalid Grasping RPC usage
_generate_grasps_for_pick() calls self.get_rpc_calls("GraspingModule.get_latest_grasps"), but rpc_calls only declares GraspingModule.generate_grasps (manipulation_module.py:120-123) and GraspingModule doesn’t implement any get_latest_grasps RPC (it publishes to an Out[PoseArray] port instead; see dimos/manipulation/grasping/grasping.py:37-106). This will raise ValueError at runtime on the “GraspGen succeeded” path.
Additional Comments (1)
This switched to |
mustafab0
changed the title
1 participant
Problem
ManipulationModule had no skill layer — only low-level RPCs requiring a custom IPython client.
No way for an LLM agent to invoke pick/place/move skills through the framework.
RRT planner rejected valid joint states due to floating-point drift past URDF limits.
Solution
Added 11
@skill()methods to ManipulationModule (pick, place, move_to_pose, scan_objects, etc.).Changed base class to
SkillModuleso skills auto-register with agents viaautoconnect().Created
xarm-perception-agentblueprint composing xarm_perception + llm_agent + human_input.Added detection snapshot so pick uses stable labels instead of volatile live cache.
Added
limit_eps=1e-3tolerance to RRT planner joint limit validation.Removed ManipulationClient and
run_*RPC wrappers — agent CLI replaces them.CoordinatorClient updated to route execution through
task_invokeinstead of removed RPCs.Breaking Changes
ManipulationClientdeleted — usedimos run xarm-perception-agentor direct RPC.run_pick/run_place/run_place_back/run_go_initRPCs removed from ManipulationModule.How to Test
dimos run coordinator-mockthendimos run xarm-perception-agentscan the scene— verify objects listed with 3D positionspick up the <object>— verify approach, grasp, retract sequenceplace it back— verify placement at original pick positiondimos agentspycloses DIM-351
closes DIM-419